Item Response Theory and Rasch Measurement Theory

Latent variable

Based on observed indicators (response data collected from participants), the latent variable is assigned a value for each respondent/participant. The value of the latent variable is the measurement. The indicators themselves are not measurements, they are indicators of a latent variable.

HS.model <- ' visual  =~ x1 + x2 + x3 
              textual =~ x4 + x5 + x6
              speed   =~ x7 + x8 + x9 '

fit <- cfa(HS.model, data = HolzingerSwineford1939)

lavaanPlot(model = fit)

Types of data

• Dichotomous data - two categories (yes/no; correct/incorrect)
• Polytomous data - more than two ordered categories (Likert etc)
• Interval data - i.e response time
• Count data - i.e frequency of behavior during a time period

We’ll primarily look at the first two types, as they are by far the most used in psychology and psychometrics.

“Sum score and alpha”

• This is a huge problem in published research.
• Many papers include “measures” that consist of questionnaires created ad hoc for a study, report Cronbach’s alpha, and use a sum score based on putting numbers on response categories.
• Most psychological “measures” are actually used only once (Elson et al. 2023)

We need a reasonable psychometric analysis to justify the use of a sum score! And even then the “sum score” is a debatable metric in itself since it is ordinal data, but often gets treated like interval data in statistical models.

Classical vs modern test theory

CTT = factor analysis, principal component analysis, etc

Modern test theory = IRT, Rasch, Mokken, etc

Major differences:

• CTT does not include analysis of item location/difficulty, which is a key aspect of IRT
• CTT assumes that reliability is constant across the continuum
  • assumes equal reliability for all respondents
• CTT has no analysis of monotonicity/ordering of response categories
• IRT enables the analysis of items and persons on the same (interval level) scale
• IRT enables estimation of latent variable scores on an interval scale
• Only Rasch models can justify ordinal sum scores as a sufficient metric to represent measurement on the latent construct scale
  • Also… Rasch is a special case, see (Andrich 2004; Kyngdon 2008).

Ability testing

• As a first and simple example, we’ll look at data from a test where a participant can either score correct (1) or incorrect (0). We will use the dichotomous Rasch model.
• A key assumption of the Rasch model (shared with all IRT models) is that items will have a systematic ordering of difficulty that is similar across participants.
• This is the structure of the dataset (items = columns):

Guttman pattern

The assumed basic structure of the data is that there is a systematic pattern across items and participants of an increased probability of correct responses as the latent ability increases. The Rasch/IRT 1PL model can be described as a probabilistic Guttman scale (Andrich 1985).

RIheatmap(rasch_data)

This figure shows items and persons sorted based on the number of correct responses (colored blue). You can see the gradual shift from lower left to upper right that shows the Guttman pattern.

Item difficulty/location

rasch <- RM(rasch_data)
plotICC(rasch, item.subset = 4, ask = FALSE, xlim = c(-5,6))
abline(h = 0.5, lty = "dashed")

This is a key figure in understanding IRT and the concept of item difficulty/location. The point on the x-axis where the line crosses 0.5 is the item difficulty (item location). This is the threshold when the probability of a correct response becomes higher than the probability of an incorrect response. ICC = item characteristic curve.

Item difficulty pt 2

Nine dichotomous items in the same plot.

plotjointICC(rasch, xlim = c(-5,5))
abline(h = 0.5, lty = "dashed")

This figure illustrates how the items are ordered (“item hierarchy”) and the locations on the latent variable where they provide the most information - the “item threshold” - which is the location of probability 0.5 for each dichotomous item.

Targeting

RItargeting(rasch_data, dich = TRUE, xlim = c(-4,4))

Targeting pt 2

Summing up so far

We have looked at the Rasch model for dichotomous data, also sometimes (not quite correctly) referred to as the IRT 1PL model. The key parameter is the item location/difficulty. 1PL stands for “one parameter logistic” model, and the parameter is the item location.

I generally use the term “location” for both items and persons as it is more generic. But for didactic purposes, when speaking of ability tests, it is probably easier to think about the specific term “difficulty” for items and how it related to the persons latent “ability”.

In IRT terminology, “person location” is frequently referred to as “theta”, often using this symbol: \(\theta\)

A note on scaling

IRT/Rasch uses the logit scale, which is an interval scale, for both items and persons. This means that the distance between two points on the scale is the same, regardless of where on the scale you are.

However, the values on the logit scale have no inherent meaning or external reference point. This is why we need to look at the item difficulty and person ability in relation to each other. Do not conflate the zero point on the logit scale with something like 0 on a Z-score scale.

Other IRT models

The 2PL and 3PL models are also commonly used. The 2PL model adds a second parameter, the item discrimination, which is a measure of how well the item separates between persons with high and low ability.

mirt(rasch_data, 1, itemtype = "Rasch", verbose = FALSE) %>%  plot(type="trace", facet_items = FALSE)
mirt(rasch_data, 1, itemtype = "2PL", verbose = FALSE) %>%   plot(type="trace", facet_items = FALSE)

3PL model

The 3PL model adds a third parameter, which makes the ICC figure look like this for example.

mirt(rasch_data, 1, itemtype = "3PL", verbose = FALSE) %>%   plot(type="trace", facet_items = FALSE)

Can you guess what the third parameter is?

ICC for polytomous models

This is the same type of figure as before, and it now shows probabilities for all response categories for one item. How do you interpret it?

Guesstimating theta

Recall \(\theta\) ? Let’s say a person responds “Often” to this item, in which range is it most likely that their theta is?

Guesstimating theta

This is an item from the Perceived Stress Scale (PSS).

Estimating theta

In context

Is this person’s theta a high or low value? How does it relate to the overall population?

Adding another item

q8n: “found that you could not cope with all the things that you had to do?” - “Sometimes”

One more item…

q2n: “felt that you were unable to control important things in your life?” - “Seldom”

See how that error bar gets smaller and smaller as we add more items? That’s because we’re getting more and more information about the respondents theta with each item.

PSS-7 targeting

Here are all 7 items from the PSS negative items (Rozental, Forsström, and Johansson 2023). Discuss 2 & 2 how you interpret this figure.

RItargeting(pss7, xlim = c(-4,4))

PSS-7 item hierarchy

Why is this figure of interest?

RIitemHierarchy2(pss7) + theme_rise()

Reliability

Test information - reflects item properties, not sample/person properties.

• The same goes for test-retest as a reliability test, it should be used to assess stability in item properties, not sample properties.

RItif(pss7) + theme_rise()

Unidimensionality

While multidimensional constructs are possible, it is outside the scope of this lecture. Most often, even unidimensional measures are not well constructed. The most common problem (imho) is residual correlations. We’ll look at four ways to assess unidimensionality in a Rasch model:

• Item fit statistics (MSQ and ZSTD)
• PCA of residuals
• Plot of factor loadings on the first residual contrast
• Residual correlation matrix for all items

PSS-14 example

We’ll use the same published dataset and paper analyzing the PSS-14 scale as before (Rozental, Forsström, and Johansson 2023) as an example for dimensionality analysis.

We use multiple tests since there is no single test to establish unidimensionality. This is also true for CTT analysis.

Does everyone know what residuals are?

Unidimensionality - Item fit

PCA of residuals

The highest eigenvalue should be below 2.0

Loadings on 1st contrast

Residual correlation matrix

We clearly have two clusters of items. Can you spot the pattern?

Ordered response categories

A higher person location on the latent variable should entail an increased probability of a higher response (category) for all items and vice versa. This is sometimes referred to as ‘monotonicity’.

We can check this by looking at the item characteristic curves (ICC). So far, we have only seen ICCs with ordered response categories. We will look at an example with disordered response categories.

An important note on types of data

• Ordinal data - ordered response categories with unknown distance between categories.

This is by far the most common type of data in psychology and social sciences. But it is extremely common to pretend that ordinal data is interval data, and use it as such in everything from simple calculations such as mean/SD to more complex statistical models. This is a problem (Liddell and Kruschke 2018) and there are methods for analyzing ordinal data properly (Bürkner and Vuorre 2019).

Adding more response categories does not make them anything else than ordinal, neither does removing the labels. Visual Analogue Scales have no strong case for being interval level either. All three methods usually add to problems with disordered response categories.

Example: Flourishing Scale (FS)

We’ll use an open dataset (Didino et al. 2019) of the Flourishing Scale (Diener et al. 2010). It has 8 items:

FS response categories

All items share the same set of 7 response categories: